1. Field of the Invention
One or more embodiments of the invention are related to the field of image analysis and image enhancement and computer graphics processing of two-dimensional images into three-dimensional images. More particularly, but not by way of limitation, one or more embodiments of the invention enable an automated and semi-automated director-style based 2D to 3D movie conversion system and method that allows for rapid conversion of a sequence of two-dimensional images into three-dimensional images.
2. Description of the Related Art
Generally, movies are shot by a director using a style that is characteristic of that director, and which does not change much over time. For example, nearly all directors use a master shot, which gives context to the viewing audience, a medium shot, that focuses on the performers, close-ups and cutaways that respectively show the actors up close or items related to the actors, such as an item that an actor is using or referring to. A scene may then end with a master shot. This basic formula has not changed over a long period of movie making.
Cameras were eventually placed on trucks for side-to-side or simulated zoom, since movie cameras originally did not have zoom lenses. Later, vertically moving platforms were introduced to allow for up and down motion of a camera, followed by jibs that allowed the camera to remain pointed at an actor while being moved up and down. Directors generally utilize these techniques and equipment to film movies in the same director-specific manner over time. Directors generally do not change their specific style of shooting a movie. For example, some famous directors utilize only a few camera moves only in a particular movie.
The vast majority of movies ever filmed have been filmed with a single lens system as opposed to a stereoscopic camera with a pair of lenses. Recent movies shot in 3D or converted into 3D from a 2D movie have generally grossed approximately 40% more than the 2D version of a movie in a given theater complex. Hence, there is a large demand for the conversion of movies from 2D to 3D. Converting movies from 2D to 3D has traditionally been time consuming, expensive and limited in the amount of movies that can be converted. The advent of home-based 3D television sets adds to the demand for 3D content. Camera systems for shooting a movie in 3D are highly technical and have many problems including un-matched CCD chips, slightly rotated lenses, slightly different lens characteristics, etc., that make for viewing a stereoscopic pair of images from the two cameras difficult for the human brain. In addition, many of the 3D camera systems have limitations with respect to the effective depth that can be captured. For example, some 3D camera systems do not allow for capturing depth within about 6 feet. This requires directors to shoot certain shots in a completely different manner in which they would normally shoot a scene. The cost and weight of these types of cameras is very high as well. Most directors simply prefer to shoot their movies with standard single lens cameras and convert the film to 3D later.
Once a movie has been converted to 3D, there are many different technologies that are utilized to view 3D images. The different viewing technologies include shutter glasses that rapidly turn one lens on and off in alteration with the second lens of a set of glasses worn by a viewer. Another technology utilizes polarized lenses, where one lens of a pair of glasses is polarized vertically and the other lens is polarized horizontally. Yet another technology utilized Red and Blue lenses and an anaglyph image that has Red and Blue images for the right and left eyes superimposed. Regardless of the type of viewing technology, the viewer is able to perceive 3D images from a flat viewing surface.
Generally, directors shooting movies with a single lens camera have struggled over time to give a sense of depth to a two-dimensional movie. For example, some directors simply utilize a bluish or teal lighting scheme in the background and an orange or reddish lighting scheme in the foreground to give a sense of depth. Other directors may frame close up objects on the left or right side of the frame to give a sense of depth to the actors in the mid-ground or background of the image. In addition, a viewer may derive a sense of depth based on the motion of objects from frame to frame. For example, objects that appear to move fast in a scene may give a sense of near-depth as these are generally objects in the foreground. In addition, far away objects may be less saturated in color due to atmospheric interference, while objects that are near may be fully saturated with color. In addition, many directors frame using the “rule of thirds”, so that visually important items, or objects of significance, such as actors, horizons and framing objects are shot at about one-third of the way from the top of the image and/or one-third of the way from the bottom of the image. In addition, another generally observed principle is that objects that are nearer are in the bottom one-third of the frame, while actors and mid-ground objects are in the middle-third, and background objects are in the upper-third of the frame.
Existing systems that are utilized to convert two-dimensional images or sequence of images that make up movies, to three-dimensional images are of one of two types. One type of conversion system is a manual conversion type. Regardless of the type of conversion, depths are assigned to human-perceived objects in a frame and these depths determine the amount of horizontal shift that is required to move objects in the frame left or right based on their assigned depth. By shifting portions of the image associated with a human-perceived object to the left, a right viewpoint image is created, by shifting portions of the image to the right, a left viewpoint image is created. Shifting objects can occur in each horizontal direction as well to effectively place objects nearer or further away from the distance implied by the captured image, i.e., the objects can be moved to or from a desired distance from an initial distance implied by the 2D image. Generally, the more shifting that occurs, the nearer the object. When the left viewpoint image is viewed by a viewer's left eye and a right viewpoint image is viewed by a viewer's right eye, stereoscopic vision occurs and the original 2D image appears as a 3D image with depth.
The manual conversion type process makes use of masks which historically have been laboriously created and manually reshaped from frame to frame to keep the mask situated on a desired area, with the correct shape and depth. This type of conversion produces very good results if diligence is observed in masking, however, the amount of labor required is extremely large. In addition, masking errors may be found late in the process and require rework by a set of workers that are geographically distant from the stereographers that utilize the masks, generally in a different time zone and/or country. This adds delays to the conversion process.
The second type of conversion system is an automated conversion type. The automated conversion type makes use of general characteristics of scenes in order to apply depth automatically in a crude fashion. For example, one type of automated conversion process applies a depth ramp to the bottom portion of a picture under the assumption that the bottom third of a picture is a floor in which mid-ground actors are standing. This type of conversion can occur in real-time, for example in a television that is programmed to show 3D images from a 2D video stream, however, the results are generally poor and may not be agreeably viewed based on what is actually in the scene. Another type of automated conversion process utilizes blue in the upper third of the image to set that part of the background to a deep distance, under the assumption that anything blue in the upper third of the image is sky. Another type of automated conversion applies closer depths to objects that are moving from frame to frame in a scene as being closer to objects that are further away. Many of these conversion techniques fail when utilized on images that do not conform to the assumed properties. For example, attempting to convert a scene of the ocean with a Blue/Red analysis does not work. Converting a panoramic scene with no floor by applying a ramp process to the bottom-third of the image also fails.
There are hundreds of types of automated processes for automatic conversion of 2D to 3D movies, but only certain types work for certain types of images. The main problem is that there is no known system that determines what processes to apply to which images, and specifically there is no known system that takes into account repeating patterns used by a particular director that can give clues as the best process or combination of processes to use to determine the depth of objects in the images that make up a movie. However, if the particular characteristics of a director were taken into account, the decision of which processes or combination of processes to use could be determined or at least narrowed down. Hence, it would be beneficial if there was a process for determining which conversion process or processes or combination thereof, to use for each frame, based on characteristics of lighting, lenses, color schemes or camera moves that a particular director uses time and time again. Hence there is a need for a director-style based 2D to 3D movie conversion system and method.